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[57] ABSTRACT 

The present invention discloses a novel system for control- 
ling the inbound and outbound data packet flow in a com- 
puter network. By controlling the packet flow in a computer 
network, private networks can be secured from outside 
attacks in addition to controlling the flow of packets from 
within the private network to the outside world. A user 
generates a rule base which is then converted into a set of 
filter language instruction. Each rule in the rule base 
includes a source, destination, service, whether to accept or 
reject the packet and whether to log the event. The set of 
filter language instructions arc installed and execute on 
inspection engines which are placed on computers acting as 
firewalls. The firewalls are positioned in the computer 
network such that all traffic to and from the network to be 
protected is forced to pass through the firewall. Thus, 
packets are filtered as they flow into and out of the network 
in accordance with the rules comprising the rule base. The 
inspection engine acts as a virtual packet filtering machine 
which determines on a packet by packet basis whether to 
reject or accept a packet. If a packet is rejected, it is dropped. 
If it is accepted, the packet may then be modified. Modifi- 
cation may include encryption, decryption, signature 
generation, signature verification or address translation. All 
modifications are performed in accordance with the contents 
of the rule base. The present invention provides additional 
security to a computer network by encrypting communica- 
tions between two firewalls between a client and a firewall. 
This permits the use of insecure public networks in con- 
structing a WAN that includes both private and public 
network segments, thus forming a virtual private network. 

25 Claims, 23 Drawing Sheets 
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SYSTEM FOR SECURING THE FLOW OF distance tnter-networking. Instead of establishing dedicated 

AND SELECTIVELY MODIFYING PACKETS lines, enterprises can communicate using the Internet as a 

IN A COMPUTER NETWORK mediator. Once connected to a local Internet provider, pri- 

vate networks can quickly connect to any destination around 
REFERENCE TO RELATED APPLICATIONS 5 the world. 

This application is a contimiatioo-in-part of U.S. patent A private network that uses some public segments is 

application Ser. No, 08/168,041, filed on Dec. 15, 1993, US. «Ued * virtual private network (VPN). A VPN is signifi- 

Pat. No. 5,606,668. canlly less expensive and more flexible than a dedicated 

private network. Each of the private networks need only be 

BACKGROUND OF THE INVENTION 10 connected to a local Internet provider. Adding new conncc- 

,. .. . , , , , , tions is simple and inexpensive. However, a major disad- 

This application relates, in general, to a method for vanUl g Cof aVP N is that U is irisecure because of its insecure 

con rolling computer network security. More specifically it ^ IntcfDct s ^ cnt dsc lQ 

relates to an easily alterable or expandable method for a. * _ j /1X .if • _t T 

, , 3 . t ... f , t . _ the following two dangers: (1) unauthorized Internet access 

computer network secunty which controls information flow ,« - 1n „„*„,„rw /u« 1, ■ ^ j /i\ 

, , t . . . • . mI ° internal enterprise networks (break-ins) and (2) eaves- 

on the network from/to external and internal destinations. . . . , . ' . . 

dropping on and lampenng the enterprise communication as 

Connectivity and security are two conflicting objectives in uev pass through the Internet 

the computing environment of most organizations^ The ^ ^ ^ fa romrmmicali ovcr lhe 

ty pi cal modern computing system is built around network i nleni et have deterred enterprises from takkg rull advantage 

communications, supp ying Iransparent access to a mulu- 20 ofV PNs. Doing busmess over the Internet (e g., transferring 

tude of services The global avaflabihty of these services 1S ^ and ^ ^ ^ 

perhaps the single most important feature of modem com- A j ,- ■ j. . \ • i j *r • 

r .* * , 6 r» j r L r a n d delivering products) requires a reliable and effective 

puting solutions. Demand for connectivity comes both from security solution 
within organizations and from outside them. 

Protecting network services from unauthorized usage is of SUMMARY OF THE INVENTION 

paramount importance to any organization. UNIX A ,. , ,. , . , 

workstations, for example, once connected to the Internet, . Accordl ^^ hc P«sent mvention seeks to provide an 

will offer all the services which it offers another station on '^T f ?■ ea ^-' ltenble % ^ m K cth ° d wh f ch 

the next table to the entire world. Using current technology, «>^ols mformation flow on a computer network to that 

an organization must give up much of its connectivity in 30 t^^Z'Z^ P ° aPPUCaU ° n 

order to prevent vulnerability, even to the extent of elimi- ' : U0/ ^ Da ' U41 - 

nating all connections to the outside world or other sites. ***** of thc invention js to control information 

A , r . j • . . now on the network from/to internal as well as external 

As he need for increased secunty grows, the means of dcstinalions whcrc ^ rontrol inchldes al Icasl one of lhc 

controlhng access to network resources has become an a ^ information Md raodif y thc mm arjd/or 

administrative priority. In order to save cost and maintain 33 desunation address 

productivity, access control must be simple to configure and , . . r .. . .. 

transparent to users and applications. The minimization of . Yc [ ""J* ob J ecl °/ mc * 10 c ° ntr °l inforraa - 

setup costs and down lime are also important factors. Uon flow bv "J™ a P ackct fi ! tcr <*P*Me of exammg 

„ , , c , t . . , , . . ,. every packet of information flowing past a node in thc 

Pactoifihermg isame^ system, the packet being encrypted. 

provides security by controlling the traffic being passed, thus A - , , . . . . 

prevenUng illegal communication attempts, both within n t^ 1 ^ ^ ct °^ invention is to cont r^ information 

single networks and between connected networks. fl ? w bv . thc P adcct , filtcr * hc / cm thc P ackcl filt f 15 "P ab c 

« ... ... r » of passing the packet only if it is preauthorized, preferably 

Curren implementation of packet filtering aUows spec,- ^ a n( f ndcs!niclive Nudity check. 

fication of access list tables according to a fixed format. This « . . . . , L . 

method is limited in its flexibility to express a given orga- Anot ? er , of ^ e J"™ 1 *™ is to provide a genenc 

nizaUon's security pohcy. It is also limited to the set of P"** 1 filtcr module whlch 15 ^DtroUed by a set of instruc- 

protocols and services defined in that particular table. This ? ons v to im P lcmen , 1 a P 0 "^ •» a ^^e to accept 

method does not allow the introduction of different protocols (P«s) or reject (drop) the packet wherein the packet is 

or services which are not specified in the original table. 50 P™* onl y lf lt P assa ee is preauthorized. 

Another method of implementing packet filtering is tai- ^ V? thcr of thc invcnu '°n 10 P rovidc a security 

bring the computer- operating system code manually in mc,hod for a com P ulcr ne,wor k wh ' ch 15 casil y Arable by 

every strategic point in the organization. This method is 'V 8 *™ administrator without thc need to change the 

limited by its flexibility to future changes in network naturc of ^ P ackel fiIlcr ltself or 10 Wlte exlensiv e c^*- 

topobgy, new protocols, enhanced services and to future 55 Another object of the invention is to provide an improved 

security threats. It requires a large amount of work by connection validity check. 

experts modifying proprietary computer programs, making Yet another object of the invention is to provide the ability 

it insufficient and expensive to setup and maintain. to modify the packet by any of encrypting it, modifying a 

In addition, the need for secure long distance communi- destination address, accepting external inputs as criteria for 

cations between enterprises, branch ofiices and business 60 accepting, rejecting or modifying the network communica- 

partners is becoming an essential requirement in modern day U0D - 

business practice. Historically, dedicated point-to-point con- Another object of the present invention is to provide an 

nections between networks were employed for fully private encryption scheme for securing thc flow of data over inse- 

intcr-enterprise commerce and long distance transactions. cure public networks, such as the Internet, thus forming a 

However, their inflexibility and prohibitive cost have pre- 65 VPN. 

vented their widespread use. Public networks such as the According to an aspect of the present invention, there is 

Internet provide a flexible and inexpensive solution for long provided a computer system to secure transactions over 
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networks by encrypting them, inter-connect various net- and selectively modifying the data packets n the computer 

w^^th o^Sadlessing schemes and provides ways network in accordance With a security rule, where each 

rolL^ackTSrmaUon only when the source of the aspect of the computer network inspected by the security 

% auZJSd and detecting the validity of rule has been previously defined, the security rule being 
£ffi"gh the netoTwhUe .Sg the infoL- 5 previously defined in terms of the aspects ar«l converted into 

UoTrS to achieve it, preferably in a fail-safe archi- packet filter language ^strucUons a method for operaUog 

uon requueu w " ^ secarity system including the steps of providing a packet 
tecture. „ r , . ^ mKn filter module coupled to the computer network in at least one 

Tnere is provided in accordance wim a preferred embody J ^ ± ^ ^ by mc 

ment of the present invention a method of inspecting and ^ * emulating a virtual 

selectively modifying inbound and outbound data p^efcin to ™ fi £^^^^ 

a computer network, the inspection and selective modifica- V™ * pass inginto and out of the computer 

Uon of the data packets occurring accordance ; wit h a g P mtermSdule reading and executing the 

security rule, the method including the steps of generating a «i fo t performing packet 

definition of each aspect of the computer network inspected J,,^ „ wa SiBtarinfi the icnlls obtadned in the step of 

bymes^tynde.gerKraUngfcese niter language instructions 

the aspect deflations, the secunty nile ccntroUmg at least °g ^i^md the packet filter module uUlizing the 

one of the aspects, converting the security rule into a set of * ^ ^ £ 1 ^ for ^ ^ 

packet filter language nistnictions for coptrolUng an opera- ^ ^ * o ^ jcct thc gc of ^ 

tion of a packet filtering module which ms^ctsand sel^- P * computer network and to 

tively modifies the data packets in accordance with fce ™ a J££* ^ ^ daU ^ J acceptcd . 

selectively modify the data packets so M ? ously defincd> the security rule being previously 
Further, the aspects can mclude network objects, network £ rf ^ ^ ^ veiled into packet 
services or network services In addition, the object dfefim- languag< , instructions, the security system including a 
tions include the address of (he object and the filter language p ^,^j module coupled to the computer network, the 
instructions of the step of converting are in the form of script f ^ ^ modu]c opcraling m accordance with the sccu- 
and further comprise a compiler to compile the script into ^ v ^ t ^ modulc implcmCDting a virtU al 
the instructions executed in the step of executing. ^ fiUcring machine inspecting and selectively modify- 
Still further, both the steps of generating the aspects of the . m& (JaU paclccts passing into and out of the computer 
network and of the security rule are defined graphically and ^wo^ and processing means for reading and executing 
the selective modification is chosen from the group consist- ^ packcl language instruction integral with the packet 
ing of encryption, decryption, signature generation and w filtcr m^i^ processing means operating the packet 
signature verification. filtering module to cither accept or reject the passage of the 
There is also provided in accordance with a preferred packets into and out of the computer network and to selec- 
embodiment of the present invention, in a security system uve i y modify the data packets so accepted, 
for inspecUng and selectively modifying inbound and out- DESCRIPTION OF THE DRAWINGS 
bound data packets in a computer network, the secunty 4 5 D 

system inspecting and selectively modifying the data pack- FIG. 1 is an example of a network topology; 

ets in the computer network in accordance with a security piG. 2 shows a security system of the present invention 

rule, where each aspect of the computer network inspected applied to the network topology of FIG. 1; 

by the security rule has been previously defined, the security pjQ 3 shows the computer screen of the network admin- 
rule being previously defined in terms of the aspects and so istrator of FIG. 2 in greater detail; 

converted into packet filter language instructions, a method pj G 4 ^ a flow diagram 0 f the subsystem for converting 

for operating the security system including the steps of grap hical information to filter script; 

providing a packet filter module couple to the computer nG 5 fa a flQW diagram 0 f an information fiow on a 

network in at least one entity of the computer network to be ^pu^r nctW ork employing the present invention; 
inspected by the security rule, the packet filter module 55 nG 6 ^ a flow diagram 0 f the operation of the packet 

implementing a virtual packet filtering machine inspecting fo n(J 5 . 

and selectively modifying the data P»^* Pf^**^ FIG. 7 is a flow diagram showing the virtual machine 

out of the computer network, and the packet filter module ri "; » a * 

aiming ft. £fc« hvvtate for opera,- *^-™^ 

ing the virtual packet filtering machine to either accept or w FIG. 8 is a Dow Diagram 01 mc uaia e*u 

reiect the passage of the data packets into and out of the FIG. 7; 

Smputernetworkand to selecUvely modify the data packets FIG. 9 is a flow diagram of the logical operation method 

so accented of FIG * 7; 

A]so provided in accords with a preferred embodiment FIG. 10 to a flow diagram of .be comparison operation 

ets in a computer network, the security system inspecting literal value to memory; 
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FIG. 12 is a flow diagram of a conditional branch opera- rently utilized to program routers, as is well known to those 

tion; skilled in the art. 
FIG. 13 is a Sow diagram of an arithmetic and bitwise Packet filters 204 arc also installed on the gateway 122 of 

operation; the remote site 120. One packet filter is installed on the 
FIG. 14 is a flow diagram of a lookup operation; 5 connection between the satellite 112 and the gateway 122, a 

jxn i< , n™, «f . m „j ^ rQ< - second packet filter is installed on the connection between 

FIG. 15 is a flow diagram of a record operaUon, ^ ^ ^ wd z third packet filter is 

FIG. 16 is a high level block diagram illustrating an Stalled on the connection between the gateway and the 

example configuration employing firewalls constructed in network. 

accordance with the present invention; a0 flows on lbe m ^ fom of 

HG. 17 is a high level block diagram illustrating the data ^ ^ to m the art. Hie location of 

transferred between two firewalls during a session key mc packjct m nG 2 is chosen so that data flow to or 

exchange; from a particular object of the network, such as a 

FIG. 16 is a high level logic flow diagram illustrating the workstation, router or gateway can be controlled. Thus, 

process performed by a firewall in transmitting a packet 15 workstations 104 each have a packet filter so that the 

using encryption to another firewall during a session data information flow to/from these workstations is separately 

exchange; controlled. At the remote site 120, however, the packet filter 

FIG. 19 is a high level logic flow diagram illustrating the is placed on the connection between (he gateway 122 and the 

process performed by a firewall in receiving an encrypted network, thus there is no individual control over the data 

packet from another firewall during a session data exchange; 20 flow to/from the workstations 124. If such individualized 

FIG. 20 is a high level block diagram illustrating the data control were required, packet filters could be placed on each 

transferred between two firewalls during a basic key of the workstations 124, as weU. Each of the packet filters is 

exchange- installed at the time that the network is set up or the security 

FIG. 21 is a high level block diagram illustrating an M fJf* * ^f£*P^Mtos <xnbc 

example configuration employing a client pcrsorJcW * ™<f? ? l * dat ^ c filtcre ■» ^cdonUie 

puter and a firewall constructed in accordance with the hosl dcvicc »<* ■* workstation or gateway at which 

present invention; protection is desired. 

FIG. 22 is a high level block diagram illustrating the data of * c F«** *teis operates on a set of instructions 

transferred between a client personal computer and a fire- 30 ■»» been generated by the packet filter generator 208 

wall during a session key exchange; and m ^ m admimst " tor X ° 2 ^ *f*<*f* cna ^ lc 

__ . .. . t ' ... „. , ,. . # . complex operations to be performed on the packet, rather 

HG. 23 1 isi bigb Icve block diagram illustrating the data ^ v anSaA of ^ £ct ^ a 

transferred between a client personal computer and a fire- ^ for or * jectioD 

wall during a basic key exchange. ^ of the packet. Thus, each packet filter can handle changes in 

DETAILED DESCRIPTION security rules with great flexibility as well as handle multiple 

security rules without changing the structure of (he packet 

Securing Inbound and Outbound Data Packet Flow filter itself. 

Referring now to FIG. 1, an example network topology is The system administrator enters the security rules via a 
shown. In this example, the main site 100 contains a system 40 graphical user interface (GUI) which is displayed upon the 
administrator function embodied in workstation 102. This monitor 206 and explained in more detail with respect to 
workstation is coupled to the network which includes work- FIG. 3. This information is processed by the packet filter 
stations 104, router 110 and gateway 106. Router 110 is generator 208 and the resulting code is transmitted to the 
coupled via satellite 112 to a remote site via gateway 122. appropriate packet filter or filters in the network to perform 
Gateway 106 is coupled via router 108 to the Internet. The 45 function that is desired. Control module 210 enables the 
remote site 120 comprises workstations 124 which arc system administrator to keep track of the operations of the 
coupled to the network and via gateway 122 to the Internet. network and storage 212 can be utilized to keep logs of 
The particular configuration shown herein is chosen as an operations on the network and attempts of illegal entry into 
example only and is not meant to limit the type of network the network. The system operator can thereby be provided 
on which the present invention can work. The number 50 with full reports as to the operation of the network and the 
configurations that networks can take are virtually limitless success or failure of the security rules. This enables the 
and techniques for sotting up these configurations are well security administrator to make those changes that are appro- 
known to those skilled in the art. The present invention can priate in order to maintain the security of the network 
operate on any of these possible configurations. without limiting its connectivity. 

FIG. 2 shows the network of FIG. 1 in which the present 55 FIG. 3 shows the computer screen 206 in FIG. 2 in more 
invention has been installed. In FIG, 2, elements also shown detail. The screen is broken into four windows, two smaller 
in FIG. 1 have the same reference numerals. As shown, the windows at the left side and two larger windows at the right 
system administrator 102 includes a control module 210, a side. Network objects and services are two aspects of the 
packet filter generator 208, a display 206 and a storage network which must be defined in the security method of the 
medium 212. Packet filters 204 have been installed on the 60 present invention. Window 304 is used to define network 
system administrator, workstations 104 and gateway 106. objects such as the workstations, gateways and other corn- 
Gateway 106 has two such filters, one on its connection to puter hardware connected to the system. It is also possible 
the network and one on its connection to the router 108. to group various devices together such as, for example, the 
Routers 108 and 110 each have a programming script table finance department, the research and development 
which is generated by the security system, but which forms 65 department, the directors of the company. It is thus possible 
no part of the present invention, and will not be described in to control data flow not only to individual computers on the 
detail. These tables correspond to the tables that are cur- network, but also to groups of computers on the network by 
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the appropriate placement of packet filters. This allows the 
system operator have a great deal of flexibility in the 
managing of communications on the network. It is possible 
for example to have the chief financial officer as well as 
other higher ranking officials of the company such as the 
CEO and the directors able to communicate directly with the 
finance group, but filter out communications /rom other 
groups. It is also possible to allow electronic mail from all 
groups but to limit other requests for information to a 



to practice the present invention. The system snapshot 
displays a summary of the system using graphical symbols. 
The summary can include, for example, the host icon, host 
name, rule base name, which is the name of the file con- 
taining the rule base, and the date the rule base was installed 
on the host. It can also show the status of the host indicating 
whether or not there have been communications with the 
host as well as the number of packets inspected by, dropped 
and logged by the host. 



groups but to limit omer requests for information to a FIG. 4 shows a flow chart of the subsystem for converting 

specified set of computers. This allows the system operator w ^ ^formation on me G UI to a filter script which contains 

to provide internal as well as external secunty for the me n|les utilizcd for ^ packet filter. In the preferred 

4 J, TTi. nhufi itrfim'h'nn ti/cttilH include ihe address of . .. »t_ _ cn • * * 



network. The object definition would include the address of 
the object on the network, as well as a name or group 
whether the object is internal or external to the network, 
whether or not a packet filter has been installed on this object 
and a graphical symbol. The graphical symbol is used in 
connection with the rule base manager 302. 

Similarly, network services are defined in block 306 on 
the screen. These network services can include login, route, 
syslog and telnet, for example. Each service is defined by 



15 



u*v i-iw «w t — preferred 

embodiment, the output of the filter script generator is 
compiled into object code which is then implemented by the 
packet filler module, as described below. 

The subsystem 400 starts at 402, proceeds to block 404 
which is obtains the first rule from the GUI. The first rule is 
the first line on the screen in which a new security rule has 
been identified, as shown in FIG. 3. Control then proceeds 
to block 406 in which code is generated to match the rule 
source network objects. That is, the source of the packet is 

* i • i_ it _ j_ ui~~i, MM «<-. n <; nn am nf 



generic and specific properties. Hie generic properties 20 entered into the source code block as representing one of 
r. «u_ ~^a. e «nnn that tVL-nti'fi j-c the service, for nh;«4c of the sv&tem from which the data, oacket will 



include the code string that identifies the service, for 
example 'dport' (destination port) which is equal to 23 for 
telnet The code string that identifies the incoming and 
outgoing packets are identified. Specific properties include 

-1. _/? it_^ * ^ 4lV_ In «HiAtn^* *W* c^nTlftft 



objects of the system from which the data packet will 
emanate. Control then passes to block 408 in which code is 
generated in the destination code block to indicate which 
object of the network the data packet is destined for. Control 



the name of the service, the port used to provide the service, then passes to block 410 in which code is generated to match 
the timeout in seconds of how long a connectionless session 25 the rule services that were chosen. The rule services have 

• . » At. j. 1 " _ AJ J.>1 ' - - - 9 2 «* I . . _1 _ _C _^ . J - 1 mt ^mmJt & A-k «*4***<ISfcj4 IHl i t^«n #tl -ft Cl^rf *FT1 f^V 



may stay inactive, that is, having no packet transmitted in 
either direction before assuming that the session is com- 
pleted. Other elements of a service definition might include 
the program number for RPC services and the outbound 



been defined previously and are stored within the system or, 
if not defined, will be defined at the time the secunty rule 
regulating the service is entered into the system. Control 
then passes to block 412 in which code is generated to accept 

' . j-.- 1-1 1. Att£ Ana .«i x-ift «»« 



me program numocr ior ivrv, «tviv»» nw w uutwuuu msu w ww* m muvu wv u jvibk-™ ~ . r - 

connections for accepted services that use connectionless 30 or reject the packet if the data block 406, 408 and 410 were 

. t l- Ttrvn *n— - -V,."., ~»>kn1 ittr mlnr or* ~,~*~U*.A «k«t ic- Iti- wmiltc nf the rrher-W were tnie. The 



protocols such UDR The graphic symbol and its color are 
specified. 

Block 302 is the rule base manager which allows the new 
security rule to be entered into the system in a graphical 
manner, thus freeing the system administrator from having 
to write code to implement a particular security rule or to 
change a security rule. Only four elements are required to 
enter the new security rule into the system. The first element 
is the source of the data packet and the third element is the 
destination of the packet The second element is the type of 



35 



matched, that is, the results of the checks were true. The 
action to accept or reject is based upon the action chosen in 
the security rule. Control then passes to the decision block 
414 which determines whether or not more rules are to be 
entered into the system. If no more rules are to be entered 
into the system, the subsystem terminates at block 418. If 
more rules are to be entered into the system, control passes 
to block 416 which obtains the next rule and passes control 
back to block 406 at which time the process repeats and the 
next security rule, found on the next line the GUI is 



ISO MODEL 



service that is involved and the fourth element is the action 40 processea 
that should be taken. The action that can be taken includes Communication protocols are hyered, which * also 
acceptmepacketmwhichcasemepacketispassedfromthe referred as a protocol stack. The ISO (International Stan- 
source to die destination or reject the packet in which case dardization Organization) has defined a general model 
the source is not passed from the source to the destination. which provides a framework for design of communication 
Ifme P acketisrejected,noactioncanbetakenoranegative 45 protocol layers. This model serves as a basic reference for 
acknowledgment can be sent indicating that the packet was understanding the functionality of existing communication 
not passed to the destination. In addition, a further element protocols, 
which can be specified is the installation location for the rule 
which specifies on which objects the rule will be enforced 
(see FIG. 2). If an installation location is not specified, the 50 
system places the packet filter module on the communica- 
tion destination by default These objects are not necessarily 
the destination. For example, a communication from the 
Internet and destined for a local host must necessarily pass 
through a gateway. Therefore, it is possible to enforce the 
rule on the gateway, even (hough the gateway is neither the 
source nor the destination. By entering the data with acro- 
nyms or graphic symbols, each rule can quickly be entered 
and verified without the need for writing, compiling and 
checking new code for this purpose. Thus, the system 
administrator need not be an expert in progra mm i ng a 
computer for security purposes. As long as the service is one 
of the services already entered into (he system, the computer 
serving as the host for (he system administrator function will 
process the information into a set of instructions for the 
appropriate packet filter, as described in greater detail below. 

Block 308 is a system snapshot which summarizes the 
setup and operations of the security system. It is not required 



55 
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Layer 


Fimctioa»lity 


Example 


7 


Application 


Telnet, NFS, Novell NCP 


6 


Presentation 


XDR 


5 


Session 


RPC 


4 


Transport 


TCP, Novel SPX 


3 


Network 


fP, Novell IPX 


2 


Date link (Hsrdw*ie Interface) 




1 


Physical (Hardware Connection) 





Different communication protocols employ different lev- 
els of the ISO modeL A protocol in a certain layer may not 
be aware to protocols employed at other layers. This is an 
important factor when making security actions. For 
example, an application (Level 7) may not be able to identify 
the source computer for a communication attempt (Levels 
2-3), and therefore, may not be able to provide sufficient 
security. 
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FIG. 5 shows how a filter packet module of the present 
invention is utilized within the ISO model. The communi- 
cation layers of the ISO model are shown at 502 at the left 
hand portion of FIG. 5. Level 1, block 504, is the hardware 
connection of the network which may be the wire used to 
connect the various objects of the network. The second level, 
block 506 in FIG. 5 is the network interface hardware which 
is located in each computer on the network. The packet filter 
module of the present invention intercedes between this 
level and level 3 which is the network software. Briefly, for 
the sake of completeness, the other levels of the ISO model 
are level 4, block 510 which relates to the delivery of data 
from one segment to the next, level 5, block 512, synchro- 
nizes the opening and closing of a "session" on the network. 
Level 6, block 514 relates to the changing of data between 
various computers on the network, and level 7, block 516 is 
the application program. 

Apacket entering the computer on which the packet filter 
module resides passes through layers 1 and 2 and then is 
diverted to the packet filter 520, shown on the right hand 
portion of FIG. 5. The packet is received in block 522. In 
block 524, the packet is compared with the security rule and 
a determination is made as to whether or cot the packet 
matches the rule. If the packet matches the rule, it nuy be 
logged on the system administrator's log and, if an illegal 
attempt has been made to enter the system, an alert may be 
issued. Control then passes to block 534 in which a decision 
is made whether or not to pass the packet based upon the 
requirements of the security rule. If the decision is to pass 
the packet, the packet is then passed to level 3, block 508. 
If a decision is not to pass the packet, a negative acknowl- 
edgment (NACK) is sent at block 528, if this option has been 
chosen, and control passes to clock 530 where the packet is 
dropped, that is, it is not passed to its destination. Similarly, 
if an application generates a packet which is to be sent to 
another destination, the packet leaves the ISO model at level 
3, block 508 and enters block 522 and proceeds by an 
identical process except that if the packet is to be passed it 
is passed to level 2, block 306 and not level 3, block 508. On 
level 2, the packet is then sent onto the network at block 504, 
level 1. If the packet does not match the rule, the next rule 
will be retrieved and the packet examined to see if it matches 
this rule. A default rule is provided which matches any 
packet regardless of the source destination or service speci- 
fied. This "empty rule" only has an action, which is to drop 
the packet If no other rule is matched, this rule will be 
retrieved and will be effective to drop the packet. Dropping 
the packet is the safest step to take under these circum- 
stances. The "empty rule" could, of course, be written to 
pass the packet. 

Referring to FIG. 6, 600 is a detailed description of the 
block 520 of FIG. 5..The generalized description in FIG. 6 
and the more detailed descriptions shown in FIGS. 7-10 
comprise a definition of the term "packet filter module" as 
the term is utilized herein. The capabilities shown in those 
figures are the minimal capabilities for the packet filter 
module to operate. FIGS. 11-15 show addition features 
which may also be included in the packet filter module, but 
are not required in the minimal definition of the term. 

The packet filter module is embodied in a "virtual 
machine", which, for the purposes of this application, may 
be defined as an emulation of the machine shown in FIGS. 
6-10 residing in the host computer, which is a computer on 
the network. 

The virtual machine starts at block 602 in which the 
packet is received, which corresponds to block 522 of FIG. 
5. Control passes to block 604 in which the filter operations 



are obtained from the instruction a memory (not shown). 
These filter operations are the filter operations that have 
been generated by the packet filter generator 208 shown in 
FIG. 2. Control then passes to block 604 in which the filter 
operations are obtained and then to block 606 in which the 
memory 618 is initialized. In block 608, the first virtual 
machine operation is obtained and performed in block 610. 
The virtual machine contains a memory mechanism such as 
a stack or register 618 which may be utilized to store 
intermediate values. The utilization of this stack or register 
is shown in greater detail in connection with the table shown 
below. Control then passes to decision block 614 in which 
it is determined whether or not the stop state has been 
reached. If the stop state has been reached, the decision will 
have been made to accept or reject the packet, which 
decision is implemented at block 616. If the packet has been 
passed, the packet will proceed as shown in FIG. 5. If the 
packet is rejected, it will be dropped and a negative 
acknowledgment may be sent as shown in blocks 528 and 
530. If the stop state has not been reached in block 614, the 
next operation is obtained in block 616 and the process 
repeats starting with block 610. 

The type of operations that can be performed in step 5, 
block 610 are shown more clearly in FIG. 7. In FIG. 7, block 
610 and block 614 are identical to the blocks shown in FIG. 
6. Connection 613 is interrupted by three operations which 
are shown in parallel. For the operation that is to be 
performed in block 610, control will pass to the appropriate 
block 702, 704 or 706 in which that task will be performed. 
In block 702 data extraction will be performed, in block 704 
logical operations will be performed and in block 706 a 
comparison operation will be performed. As shown at the 
right hand portion of FIG. 7, other blocks can be added in 
parallel to the operations capable of being performed by the 
virtual machine. The subset shown as blocks 702, 704 and 
706 arc the essential elements of the virtual machine of the 
present invention. These elements are shown in greater 
detail in FIGS. 8, 9 and 10, respectively. Additional elements 
which may optionally be included in the operations capable 
of being performed by the virtual machine are shown in 
FIGS, 11-15, respectively. 

The data extraction block 702 is shown in greater detail in 
FIG. 8. The process starts at block 802 and control passes to 
block 804 in which data is extracted from a specific address 
within the packet 806. This address is taken from the stack 
memory 618 or from the instruction code. The amount of 
data extracted is also determined by the stack memory or the 
instruction code. The extracted data is put into the memory 
stack 810 at block 808. The process terminates at block 812. 
In these figures, control flow is shown by arrows having a 
single line whereas data flow is shown by arrows having 
double lines. 

FIG. 9 shows logical operation 704 in greater detail. The 
logical operation starts at block 902 and control passes to 
block 904 in which the first value is obtained from the 
memory 906. In block 908 a second value is obtained from 
the memory and the logical operation is performed in block 
910. If (he logical operation is true, a one is placed in the 
memory 906 at block 912 and if the logical operation is 
false, a zero is placed in the memory 906 at block 914. The 
process terminates at block 916. 

The third and last required operation for the virtual 
machine is shown in greater detail in FIG. 10. The com- 
parison operation, block 706, starts at block 1002 and 
65 control passes to block 1004 in which the first value is 
obtained from memory 1006. Control passes to block 1008 
in which a second value is obtained from memory 1006. A 
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comparison operation between the first and second values 
takes place at block 1010. If the comparison operation is 
true, a one is placed in memory 1006 at block 1012 and if 
the comparison operation is false a zero is placed in memory 
1006 at block 1014. The process terminates in block 1016. 

The following operations are not shown in FIG. 7 but may 
be added at the right side of the figure at the broken lines and 
are connected in the same manner as blocks 702, 704 and 
706, that is, in parallel. FIG. 11 shows the entering of a literal 



described utilizing as an example the security rule to disal- 
low any Telnet services in the system. Telnet is defined as 
being a TCP service and having a specific TCP destination 
port It will be identified by having a TCP protocol value of 
6 in byte location 9 of the packet and by having a destination 
Telnet protocol number of 23 in byte location 22 of the 
packet, the value being a two-byte value. This is found in 
every Telnet request packet. 



20 



Packet 
FflUi Code 



Prop Telnet Frocctt 



Virtual Machine Operation 



Memory 
Values 
(Stack Order) 



1 23 



23 23 



i parallel, tiu. xx snows uic cuicnag ui « u*i « -j^ c operation in the table shown below is to extract 
value into the memory. The process starts at block 1102 and J0 ^ Jp pTQtoco \ from the packet location 9 and place this in 
control passes to block 1106 in which the literal value is memory As shown m the "Memory Values" column at the 
obtained from the instruction code. Hie value is placed into &idc of ^ tab , mis vahlC( 6> & placed at the top of 

the memory at block 1108 and the process ends at block stack. 

W10. The second operation, the TCP protocol (port) number, 

A conditional branch operation is shown in FIG* 12. ine JS whichisstatedtobc6abovC(is piaced at the second location 
process starts at block 1202 and control passes to block 1204 ^ me ^ stc 3 ^ vahics of mc first two layers of the 
in which the branch condition, taken from the instruction compared, obtaining a positive result- 

code, is checked. If the branch condition is true, the value is * 

obtained from the memory stack 1206 at block 1208 and 

checked at block 1210. If the results of the comparison in 
block 1210 is true, the next step is set to N and the process 
terminates at block 1216. If the comparison in block 1210 is 
false, the process terminates at block 1216. If the branch 
condition is false, at block 1204, control passes directly to 
block 1214. 

An arithmetic or bitwise operation is shown in FIG. 13. 
The process starts at block 1302 and control passes to block 
1304 in which the first value is obtained from memory 1306. 
The second value is obtained from memory 1306 at block 
1308 and an arithmetic or bitwise operation is performed on 30 
the two values obtained from the memory 1306 in block 
1310. The result of the arithmetic or bitwise operation is 
placed in the memory in block 1312 and the process termi- 
nates in block 1314. 

FIG. 14 illustrates a lockup operation which is useful if 35 
data needs to passed from a first set of instructions imple- 
menting a security rule to a second set of instructions for a 
second security rule. As shown in block 606 of FIG. 6, the 
memory is initialized whenever a new security rule is 
processed. Therefore, information placed in the memory by 40 
a first security rule will not be available for use by a second 
security rule. In order to overcome this problem, a separate 
memory 1410 is supplied which contains Tables 1-3 which va h, cs Q f g at fo c top two layers of the stack are 

can be utilized for this purpose. The entry of data into the deleted and a 1, indicative of the positive result, is placed at 
tables is shown in FIG. 15 and described below. Hie lookup 45 ^ lcp 0 j ^ stack, in step 4, the TCP protocol number for 
operation starts at 1402 and control passes to 1404 in which packct i oca tion 23 is extracted and placed in the memory 
values are obtained from memory 1406. Control passes to Nation at the second layer of the stack. In step 5, the literal 
block 1408 in which data is obtained from Tables 1-3 at value which is the Telnet protocol number is placed into the 
block 1410 by searching the values in the referred table. me mory at the third layer of the stack. In step 6, the memory 
Control passes to block 1412 in which a decision is made as 50 iay ers 2 and 3 containing the TCP protocol for Telnet is 
to whether the block is in the table. If the decision is yes, a compared with the expected value, obtaining a positive 
one is placed in memory 1406 at block 1416. If the decision result. The values of the second and third layers of the stack 
is no, a zero is placed in memory 1406 at block 1414. The m <j e i cte ,j ^ replaced by a 1, indicative of the positive 
process terminates at block 1418. result. In step 7, a logical operation is performed to see if 

Referring to FIG. 15, the process starts at block 1502 and 55 both the TCP and Telnet have been matched. This is deter- 
control passes to block 1504 in which values are obtained mmc d by a AND operation. In this case the result is positive 
from memory 1506. Control then passes to block 1508 in and the ones in the first two layers of the stack are deleted 
which values obtained from memory 1506 are placed in the ^ re placed by a 1 indicative of the positive result. In step 
appropriate locations in Tables 1-3 at block 1510. Control a conditional branch operation is performed in which if 
passes to block 1512 in which a decision is made as to 60 the memory value is true, the program branches to the drop 

.-.rU*tU+r nr nnt ih* ctnracr* vaTlIM in the table h»S SUCCCeded. ->-i_ T_ lUC tha r«ult ic lni*> %nA frw nmonm hr&ItcheS 



^ 1 pmhbyte [9] 

2 push 6 

3 co. 



4 push* [22] 



5 posh 23 

6 eq 

7 end 

8 btrue drop 



Extract Operation: Extract IP protocol 
number from packet location 9 to 
memory 

Enter literal Value to Memory: 
Put TCP protocol number in memory 
Companion Operation: Compare IP 
protocol to TCP, obtaining a potiiive 
remit 

Extract Operation: Extract 

TCP protocol number bora packet 

location 22 to memory 

Ester literal Vahie to Memory: Put 

TELNET protocol number in memory 

Companion Operation: Compare 

TCP protocol to TELNET, obtaining 

a positive remit 

Logical Operation: Check if protocol 
both TCP and TELNET are matched 
Conditional Branch Operation: If 
memory value if true, branch to drop 
state 



whether or not the storage values in the table has succeeded. 
If the storage has succeeded a one is placed in memory 1506 
at block 1516. If the process has not succeeded, a zero is 
placed in memory 1506 at block 1514. The process termi- 
nates at block 1518. 

An example of a security rule is implemented using the 
packet filtering method of die present invention will now be 



65 



state. In this case, the result is true and the program branches 
to the drop stale in which the Telnet request is not passed. 
Thus the rule to drop Telnet has been implemented. 
Encrypting Data Row — An Introduction 
As stated earlier, long distance communications between 
enterprises, branch offices and business partners have 
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become an essential part of modem day business practice. 
Utilizing the present invention, virtual private networks 
(VPNs) can be constructed over insecure public networks 
such as the Internet to provide secure and flexible commu- 
nications. 

The modification of packets by, eg., encryption of out- 
bound packets, decryption of inbound packets, signing of 
packets or address translation is performed by the packet 
filter module. The decision whether to modify a packet is 
determined from the rule base. AD modifications, i.e., 
encryption, decryption, signing and address translation are 
performed on a selective basis in accordance with the 
contents of the rule base. For encryption, for example, to 
occur, a rule in the rule base must explicitly call for 
encryption to occur on packets which have a particular 
source, destination and service type. The encryption instruc- 
tions arc translated into the packet filter language that are 
installed and executed on the virtual packet filter machines 
in the network. 

As described previously, the packet filter module deter- 
mines whether a packet is rejected or accepted. If rejected, 
the packet is dropped. If accepted, the packet may be 
modified in a number of ways. Example of types of possible 
modifications include, but are not limited to, encryption, 
decryption and address translation. The following describes 
in detail the encryption and decryption of packets that is 
selectively performed by the packet filter module. 

Notation Used Throughout 
The following notation is used throughout this document: 
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-continued 



Term 



Definition 



10 



15 



certification a misted third party, known as a Certificate Authority 
(CA), from which one can reliably obtain a public key, 
even over an insecure communication channel, generate* 
a certificate for the public key which can be verified by 
the recipient 

digital information generated from the contents of the message 

signature itself and used by the recipient to verify the data integrity 

of the message and/or its origin 
network object a piece of hardware that is connected to a network and 

which has some interaction with the network 
gateway a network object that is connected to at least two 

networks and passes information between them 
firewall or a network object, usually a gateway or an end host, thai 
fire walled secures the flow of inbound aad outbound data packets 
network abject on a computer network and also selectively modifies data 

packets in accordance with a security rule base 



20 



25 



30 



Symbol 


Description 


g 


common root used for all Diffit-Hellman keys 


P 


common modulus osed for all DifEe-Hellman keys 


fc 


source private key 




source public key 




destination private key 




destination public key 


B 


basic key 


TB 


truncated basic key 


A 


auxiliary key 


R 


session key 


E 


session data encryption key 


I 


session data integrity key 


M 


data portion of a packet 


P 


unencrypted password 


ENC,(Y) 


encrypt Y using X as the key 


DCR^fY) 


decrypt Y using X as the key 


SIGfY) 


signature of Y 
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Definitions of Terms Used Throughout 

The following definitions are helpful in understanding the 
operation of the present invention. 



Term 



Definition 



plaintext 
cleartext 
cipbertext 
key 

encrytion 



decryption 



text (hat is not encrypted 

another term for text that is not encrypted 

encrypted text 

a piece of information known only to the sender and lie 
intended recipient 

converting the plaintext of a message into ctphertcxt in 
order to make the message unintelligible to those without 
the key 

converting ciphertcxt into plaintext using the same key 
used to encrypt the message 



55 



60 
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A high level block diagram illustrating an example con- 
figuration employing firewalls constructed in accordance 
with the present invention is shown in FIG. 16. The example 
network shown in this figure will be used to explain the 
encryption capabilities of the present invention. The net- 
work configuration shown is only for illustrative purposes 
only. Once skilled in the art can adapt the present invention 
to other network configurations as well. Both hostl and 
host2 are connected to their respective private LANs. In 
addition, firewalls 1604 is coupled to hostl through its LAN 
and firewall2 is coupled to host2 through its LAN. Both 
firewalls are coupled to a public network 1606 such as the 
Internet. It is also assumed that the public network is 
insecure and cannot be trusted. Certificate Authorityl (CA1) 
1602 functions as the certificate authority for hostl and 
firewalll. CA2 1612 functions as the certificate authority for 
host2 and firewal!2. In other embodiments, there may be 
only a single CA that serves both firewalls. In either 
embodiment, the functions of the CA remain the same. The 
only difference is which CA the firewall uses to obtain public 
keys. 

It is desired that the communications between hostl and 
host2 be secured The communications from hostl is routed 
to the Internet (i.e., the public network) via firewalll which 
acts as a firewallcd network object. Similarly, communica- 
tions from host2 is routed to the Internet via firewaI12 which 
also acts as a fire walled oetwork object In communications 
to host2, firewalll intercepts and encrypts the packets it 
receives from hostl enroute to host2. Firewall2 receives the 
encrypted packets destined for host2 and decrypts those 
packets. In the opposite directions, firewall2 encrypts the 
packets from bost2 destined for hostl. Firewalll receives the 
encrypted packets, decrypts them and passes them to hostl. 
The encryption and decryption operations performed by 
firewalll and firewall2 are transparent to hostl and host2. 

Assuming hostl initiates the session with host2, it sends 
an Internet Protocol (IP) packet to bost2. Firewalll will 
intercept the packet and determine that communications 
between hostl and host2 are to be modified in some way, 
e.g., encryption, decryption, address translation, etc. The 
decision is made separately for each connection based on 
information from all the ISO layers and based on informa- 
tion retained from previous packets. This decision process is 
termed statcful multi-layer inspection (SMLI). Each firewall 
maintains a rule base that instructs the firewall how to handle 
both inbound and outbound communications between net- 
work objects, as described in detail earlier. After determin- 
ing that communications between hostl and host2 are to be 
encrypted or digitally signed, firewalll temporarily parks the 



4/30/04, EAST Version: 2.0.0.32 



5,835,726 

15 16 

£to or signing can take place, bolh sides Deed to agree the session key R^a truncated version of the basic key B is 
on a shared key. This kcy is called the session toy £ and ^^^^ miintains a tabic 0 f bindings 
isgeneratedanewatmestartofe^^ 5 ^-^^ ta and urcwallcd network 

? objects. In addition, a firewall must have a binding between 

firewall2 is encrypted because or we use oi m IP addresses and such an object. In the configuration shown 

Internet or public network. Hie communications between IP addresses ^™ »™ "J.^ firewalll must be configured 

hostl and firewaUl and between host2 and ^ » ™' fl l ££ %£^£^.BMmi«to 

encrypted because it takes place over private LANs which » fi^ * fi^allZ. Firewalll 

can be assumed to be private and secure. know that hos* ^n^g ^ ^ ^ ^ 

Session Key Exchange— FirewaWFirewall encrypting firewalls for firewalE. The bindings and the 

Session Key txenang ^jv^ ^ fof ^ fircwaU ^ managed in 

A high level block diagram illustrating the data trans- ^ fashion by a separate management unit, 
fcrred between two firewalls during a session key exchange ]n ordcr to CQcrvp t communications between firewalls, a 
is shown in FIG. 17. The following scheme is only one fircwall must knowledge of its own basic private key 
example of implementing encryption with SMU and is not ^ ^ basic pubUc kcys of cacn firewalled network object 
meant to limit the scope of the present invention to other ft Qceds to conjmunicatc with. The basic public keys belong- 
encryption techniques. It would be obvious to one skiUedm ^ . tQ CItctnal firewalled network objects such as a firewall 
the art to adapt other encryption techniques to the SMU belonging to a business partner must also be known in order 
process to carry out the teachings of the present mvention. fof encrypted sessions to occur. This static binding of basic 
For example, in an alternative embodiment the SKIP stan- to fircwaIlcd nctW ork objects may already be estab- 

dard is utilized. To initiate the encryption of data, firewalll m a database internal to the firewall or it can be 

first sends a request packet to host2. The request packet is blained on thc fl y using the basic key exchange described 
sent to host2 and not firewall2 because firewalll may not 

know the IP address of the firewall that is responsible tor Qncc a shared secret basic key B is agreed upon 

host2. Fircwall2 intercepts this request packet and returns a bythctwo nrewa lls, it is used to encrypt the actual key used 
reply packet. The request and reply packets allow both sides f ^ ^ ^ session kcy R . The same session key 

to agree on shared session key R that will be used for all ^ R ^ uscd by both saimx ^ destination to encrypt the data 
communications to be encrypted between hostl and host2. from hostl to host2 and from host2 to hostl. 
As stated previously, only the communications between Thc clcmcnts 0 f the request from the source to the 
firewalll and firewall2 are actually encrypted. destination is shown above the right arrow in FIG. 17. The 

In general, the session key R is generated by the non- mct hod comprises one or more encryption methods 

initiator (i.e., firewall2 1608) also called the destination and 35 for encrypting the session data that the source is able to 
is sent encrypted to the initiator (Le., firewalll 1604) also pcrform ^ DES, FWZ1, RC4, RC5, IDEA, Tnpple-DES, 
called thc source. This two packet exchange must occur ctc ) The key mcthod comprises one or more encryption 
before encrypted communications can proceed. After the mcthods f or encrypting the session key R that the source is 
encrypted session is established, state information is main- ablc t0 perf orm (e.g., DES, FWZ1, RC4, RC5, IDEA, 
tained in both firewalls and thc original packet that was ^ Tr ip p i c .DES, etc.). The md method (i.e., message digest 
parked is now passed encrypted through the firewalls. The m ethod) or the message integrity method comprises one or 
same session key R is used by firewall2 to encrypt packets morc mct hods or algorithms for performing data integrity 
that are sent fiom host2 to hostl. that the source is able to perform (Le., MD5, SHA, etc.). The 

The session key exchange will now be described in more data integrity typically entails calculating a cryptographic 
detail In order to agree on a common secret session key R, 45 bash of a part of or all of the message, 
the present invention uses a 'static* Diffie-Hellman scheme. Ttc suggested source public key ID identifies via an IU 
E^TSr^-Hellmankeycomprisesaprivatepartand^ number, the basic public key that the sounx a^unes he 
pT^Ea^siae has its owrTprivate and public parts. The destination will use. likewise the sugg^ed destinaUon 
orivate kev for the source (Le., firewalll) and destination basic public key ID identifies the basic public key that the 
JuteSdS riTandD^ respectively. The public 50 sour ce^ assumes the destination will use. If there are more 
p^fo^u!^ as follows: ^ one possible firewalled network 

p the source will include multiple suggested basic public keys 

in the request packet since it does not know which of the 
s^-g»~(mo<S)p firewalled network object actually serves host2. Each sug- 

_ 55 eesled basic public kcy corresponds to a different firewalled 

network object. 

Both source and destination must know each others public The request also comprises a challenge key C wbichisa 
key for the session key exchange to work. If one side docs random bit field chosen by the source (i.e„ firewaUl) wnicn 
not know the other's public key or the key U does have is is used to thwart man in the middle attacks against the 
determined to be out of date, than a basic key exchange is ^ session key exchange or the session data itself, 
trifieered which is explained in more detail below. Bolh The destination (Le., firewaH2) receives Uie request 
sides use each other's public key to derive at the basic key packcl and based on its contents generates a reply packel I to 
B The source performs the following: be sent back to thc source. The elements of the reply packe 

V are shown above the left arrow in FIG. 17. The reply packet 

65 has a similar formal as the request packet with the exception 
B-i^im^^imoQp-g^^faoQp of thc challenge key C field replaced by a field holding the 

Similarly, the destination performs the following: encrypted session key R. Each of the cipher method, key 
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method and md method now have only one element rather 
than a list of options as in the request. The elements listed 
arc the elements chosen by the destination from the options 
listed in the request. Similarly, the chosen source basic 
public key ID and the chosen destination basic public key ID 
both comprise a single key ID representing the key ID 
chosen by Ihe destination from the option list sent in the 
request. 

The session key R that is sent in the reply actually 
comprises two keys: a session data encryption key E and a 
session data integrity key I. Thus, the session key R is 
defined as 



10 



The session key is a random stream of bytes thai is generated 
for both the cipher method (i.e., encryption method) and the 
md method or method digest method. Its length is the sum 
of the key lengths needed by the cipher method and the md 
method. Once generated, a signature of the session key is 
obtained using the chosen md method, e.g., MD5, and 
represented by SIG(R). Hie combination of session R and 
SIG(R) are then encrypted using a key formed by the 
combination of the truncated basic key TB and the challenge 
C, thus forming 



15 



20 



25 



30 



35 



which is what is sent in the reply to the source. 

The signature or hash checksum computation provides 
authentication to the source that the packet it received is 
indeed formed by an entity that knows the basic key B thus 
providing strong authentication for the reply packet. In 
addition, since the source chose the challenge key C, there 
is no possibility of replay. 

Session Data Exchange 

A high level logic flow diagram illustrating the process 
performed by a firewall in transmitting a packet using 40 
encryption to another firewall during a session data 
exchange is shown in FIG. 18. Although not shown in the 
Figures, an alternative embodiment utilizes the IPSEC stan- 
dard for performing session data exchange. As mentioned 
earlier, once the source and the destination agree on a 45 
session key R, encrypted communication between both 
firewalls can proceed. Interception of and modifications to 
the packets occur between level 2 and level 3 of the ISO 
model. Communications occurring both ways is to be 



contents of the packet itself (step 1800). Hie portions used 
depend on the type of packet and protocol (e.g., RPC, UDP, 
TCP, ICMP, etc.) and may include the following fields, for 
example, IP-ID (only a portion), RPC-XID, RPC- 
PROGNUM, TCP-SEQ, TCP-ACK, UDP-LEN, ICMP- 
TYPE, ICMP-CODE and IP-PROTO. Next, the auxiliary 
key A, session data integrity key I and the data portion of the 
packet M are placed in a buffer (step 1802). A signature is 
then generated on the contents of the buffer using the md 
method (step 1804) and expressed as 

SIGCA+I+M) 

The bits of the signature generated are then placed in the 
packet header (step 1806). Adding the signature bits to the 
packet is important for ensuring data integrity. Since the 
length of the packet is not modified some portions of the 
packet must be overwritten with the signature bits. Note that 
the signature bits are stored in the packet before the packet 
is encrypted. For TCP packets a 28 bit signature is stored as 
follows: 

the 8 LSBits of the signature replace the 8 MSBits of the 
IP-ID 

the next 16 bits are added to the TCP-CSUM field using 

l's complement arithmetic 
the next 4 bits are stored in the unused TCP-X2 nibble 
(this is optional) 
For UDP packets a 32 bit signature is stored as follows: 
the first 16 bits of the signature are added to the UDP- 
CSUM field using l's complement arithmetic; if the 
original UDP-CSUM field is zero, the UDP-SPORT 
and UDP-DPORT fields are added to the UDP-CSUM 
also using l's complement arithmetic 
the next 16 bits are stored in the UDP-LEN field 
Once the signature bits are stored in the packet, the data 
portion of the packet M is encrypted (step 1808), and can be 
expressed by 

The encryption is performed using the cipher method with a 
combination of the session data encryption key E and the 
auxiliary key A. Finally, the packet is transmitted over the 
public network (step 1810). 

A high level logic flow diagram illustrating the process 
performed by a firewall in receiving an encrypted packet 
from another firewall during a session data exchange is 



encrypted and decrypted using the same session key R. The 50 shown FIG- 19. First, in order lo verify the signature, the 

packets that arc sent out closely resemble normal TCP/IP auxiliary key A must be generated from the contents of the 

packets. The packets-do not include any information indi- packet (step 1900). Then, the packet's data portion M is 

eating whether the packets are encrypted or not and if so decrypted using the cipher method and a combination of the 

which key to use. "Hiis information only exists in the state session data encryption key E and the auxiliary key A (step 

maintained by the two firewalls. The encryption is per- 55 which can be expressed as 
formed in place without changing the length of the packet 

which serves to increase the efficiency and bandwidth of ™ NC ^ 
encrypted traffic. In general, each transmitted packet is 

divided into two parts, a cleartext part which is not Next, the signature bits are extracted from the packet header 

encrypted and a ciphertext part which is encrypted. The w (step 1904). A signature on the auxiliary key A, session data 



cleartext part includes the IP header and the TCP/UDP 
header. The rest of the packet meaning its data M is 
encrypted using a combination of the session key R and an 
auxiliary key A computed from its cleartext part. The 
process will now be described in more detail. 

The first step performed by a firewall in transmitting a 
packet is to generate an auxiliary key A from the cleartext 



65 



integrity key I and the packet data M is then generated using 
the md method (step 1906), and expressed as 

SIG(A+I+M) 

Then the two signatures arc compared with each other (step 
1908). If they match the packet is passed after replacing any 
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H,t* in the nacket that was overwritten with signature data The elements of the response by firewalll are shown 

t& n miTu*%^ P«** is above the right arrow in FIG. 20. The elements comprise the 

A~A tJJ?\ <ni\ CA public key ID, the source basic public key and the 

dropped (step 1912). jp ^ of ^ aafUstm In ^ 

Basic Key Exchange $ sigMtUfC of source basic public key is sent, which can 

As explained previously, in order to encrypt communica- be re p rescntc d by 
tions between firewalled network objects, a firewall must 
have knowledge of its own private basic key and the public 

basic keys of each firewall it needs to communicate with. siO(S^) 

known in order for encrypted sessions to occur. This static first generating an intermediate signature from the basic 

binding of basic keys to firewalls can already be established pub li c key to be sent using the md method of generating 

in a database internal to the firewall or it can be obtained on digital signatures. Then, this intermediate signature is input 

the fly using the basic key exchange. In addition, the basic to fa. RSAdecrypting function to generate the signature that 

keys may be updated on an infrequent basis to improve is fi na u y transmitted The IP address of the source (Le., 

security. The present invention provides for basic public firewalll) is included in order to verify the binding between 

keys to be obtained on the fly if they are not already in a ^ fi^ai^ i.e., firewalll, and a basic public key (S^J. 

database within the firewall. In general, a basic public key Upon rccc i pt Q f foe certificate from firewalll, firewalU 

must be obtained if the source does not have knowledge of m vejif y it n5 . |ng mc CA public key. If it verifies correctly, 

the destination's basic public key or the destination deter- w firewall 2 updates its database with the new basic public key 

mines that the destination basic public key used by the ^ firewalll. Now, the session key exchange can be com- 

source is out of date. pleted and session data can then to be communicated. 

In either case, the exchange of the basic public key is ^ c ^ ^ basic publ j c ^eys are communicated 

certified in order to be sure as to the authenticity of the between cach firewall and its CAovcr secure communica- 

Diffie-Hellman key being transmitted. Certification i of ^ channels. If there is more than a single CA the public 

messages, in general, serve to thwart man in the middle ^ Qf ^ ^ fa seQl iQ ^ dcar l0 mc othcr CA. This 

attacks against the system. message is either signed using a previous value of the CA 

The process of exchanging the basic keys wul now be c ^ of ^ ncwly obUmcd CA public key can be 

described in more detail. A high level block diagram illus- verific d by some other manual means, such as facsimile or 

trating the data transferred between two firewalls during a 30 |c j c _j K)I1B 
basic key exchange is shown in FIG. 20. Whenever any of 

the two sides recognizes that either it does not have a valid Session Key Exchange— Client/Firewall 
key for its peer or that it has an outc^Ucy it request, .he e „ u ^ growiiig business need for 
olher side to send it a certified basic key. scctss to .^rate networks. More and more 
The basic key exchange can be triggered in two ways 35 ™ working physically outside the corporate 
depending on which sMe discovers, " ^g^g ^o'rWA^ enviSent but Jed to connect to it. The 
has to be updated or exchaogedJVp^Mlwmbe the side provides Uie capability of verifying exter- 
nal discovers it does not have to b«si=^-P« „ Jusers of a system and providing encrypted communica- 
example, referriig to FIG M. a bas. «^*^^nUte ""^^^^LrorcUent^the hostsystem. 

T^o, firewaiU, opting a ^S^^^Z^^^SL 

left arrow in FIG. 20. The basfc request compr^to source J^^^E of explanation. All con,- 

^^^^^JZi^SXX » -"ns between Uie PcL the Z is routed through 

method, key method and md method. These elements are so programmed to perform the 

identical to those discussed above >»». S^S^fcI^lollita^e^««^ 

Session Key Exchange^^w^ew^. When » ta»c ^SSSSStat between itself and the firewall. Similar to 

key exchange must occur the side that wants ^ the other to comm encrypted communica- 

send it a certified key update or key sync win add la CA ^"^"^^ me pc ^ ^ fi re wall in the con- 

puMic key ID field to the remjesL This new fiekl ^ates 55 g^J*^^ To ^ ^ the firewall is 

SEsWET. t^y^L^l transparent and thinks data is coming straight from the PC 

"the reply bom Brew &L Upon receiving this The session data exchange processes for client! to firewaB 

mSage firewalll will send its basic public key S> to encryption aresimtlar 'othoseof fi 

firewaM after certifying it with the CA public key against a » lion. The diHerences he, however, in ^e jesaon key 

cerUfi«Uo7^t was made by the CA. Certification is the exchange and the basic key exchange processes. With fire- 

r^Tgenl^g "digi J signature of me basic pubhc wall to firewall session key exchange, each ^received 

LyX firewalft^CAl 1602 generates the CA public keys a different session key. A session is no only a connecUon 

for verifying firewalll's basic public keys (FIG. IS). In between two particular network objects but may mchde 

order for &iwall2 to verify .he signature, il must obtain me S5 different services between the same network object In 

CA public key from CA1. thTcertificate authority for contrast, the client initiates a session witti Uie host and I al 

la puoiic ney imui wu. y communicaUons between the client and the host during that 
firewalll. 
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session is encrypted using the same key, no matter what Once the session key is known by both the client and the 

activities or services the client requests. In addition, in firewall, the communications session can proceed between 

firewall to firewall communications, both sides have each the PC and the host via the firewall and the encrypted 

other's certified public key. In client to firewall communication between the PC and the firewall is Lranspar- 

communication f this is true only for the client, while the 5 ent to the host. In order to reduce the number of key 

server identifies the client using a name/password pair sent exchanges, the session key R is used for all encrypted 

to it by the client. connections passing through the same firewall. After a 

A high level block diagram, illustrating the data trans- predetermined time duration, e.g., several minutes, the scs- 

fened between a client personal computer and a firewall sion key R is dropped, 

during a session key exchange is shown in FIG. 22. The w ^ & E]re |^ e _ C i fcllM F irewan 
elements sent in the request by the client are shown above 

the right arrow. The elements include a name, cipher 1° contrast to firewall to firewall communications, a 

method, key method, md method, password method, source certified key exchange is only necessary to update the client 

basic public key suggested destination basic public key with the firewall's basic public key. A basic key exchange 

ID, challenge key C, encrypted password and a signature. ^ may be triggered in either of two ways. The first, if the client 

The name is used to identify the user who is currently using does not navc firewall's basic public key or, second, if 

the client. The cipher method, key method and md method the firewall determines that the basic public key used by the 

are as described earlier. The password method indicates client in the request is outdated. 

which encryption method to use in encrypting the password. The process is similar to the basic key exchange as 

The encrypted password can be express as 20 explained previously in the section entitled Basic Key 

Exchange — Firewall/Firewall. However, there are differ- 
ences as explained below. If the client realizes that it does 

EN^c.oCP) not have the firewall's basic public key, it substitutes a CA 

_ . . ... . _ t 4 . public key ID field for the destination basic public key ID 

The source basic public key S U » ** « M fieldin the request. This is shown above the top right arrow 

firewall does not maintain a hst of users and their associated m nG 33 which ^ a high M block ^ ^ Saonaiug 

basic public keys. The data ^ sent is similar to toe data ^ data tnmsferred ^Icn a c]imt ^ ter £ 

sent by fircwalU to firewall2 (FIG- 20) <s described in the a firewall during a basic key cxchange/This key ID is the ID 

section entitled Basic Key fochange-FirewaWirewall. of ^ ^ficate authority key (e.g., RSA key) by which the 

The destination basic pubhc key ID is as was described 30 ^ waflts to rcccivc ^ ^\ y l om mc 

above in the section entitled Session Key Exchange — „_ . _ „ r ' _ . ..... 

Firewall/Firewall When the firewall receives the request from the client, it 

The signature functions to ensure to the destination, the ^mmn^Jtm * c « requesting 

receiving side, that the message was not modified. The ? c **** s basic Public key or the key ID in the request 
signature is generated by taking the entire contents of the 3S *»« ^pond t 0 the firewall's basic pubhc key. The 
M „„„„„. *r :« err- t-> elements of firewall's reply is shown above the left arrow. 



request or message, represented as T in FIG. 22, except for 



the signature field/and combining T with the unencrypted ^ origjiud^ested destination basic 

password and the truncated basic public key TB, expressed f™ bU ' te >l CA k f * d«tiMtoii basic pubhc 

as the following key D^ w IP address of the destination and a signature. The 



40 



original destination basic public key is taken as is from the 
request The signature of the destination basic public key is 
SK»(r+P+TB) sent, which is represented by 

The signature is added to (he request and the request then 

sent to the firewall. siop^,) 

After receipt of the request, the firewall knows the client's 45 In a preferred embodiment, the signature is generated by 
basic public key S^. It can now generate the basic key B first generating an intermediate signature from the basic 
and the truncated basic key TB. It then can decrypt the public key to be sent using the md method of generating 
password P. Once P is known, the firewall can verify the digital signatures. Then, this intermediate signature is input 
signature in the request. The firewall next generates a to the RSA decrypting function to generate the signature that 
random session key R and encrypts R and the signature of 50 is finally transmitted. The IP address of the destination (Le., 
R using the truncated basic key TB and the challenge C sent the firewall) is included in order to verify the binding 
in the request from the client, and given by between the firewall, and a basic pubhc key (Dp«fc)- 

Upon receipt of the certificate from the fircwaJl^lnc client 
can verify it using the CA public key. If it verifies correctly, 
ENq„ 4<J (R+sictR)) 55 ihe client updates its database with the new basic public key 

A signature is then generated of the content of the request of firewa ? L . 

denoted by U in FIG. 22 in combination with the truncated ^ reccmn S * c wal1 f thc c iif Dt f nds back , 
basic public key TB, as given by a mcssa 8 e 10 ""P 1 * 1 * mc ^IhcnbcaUon. The elements of 

the message are shown above the bottom right arrow in FIG. 
6° 23. The message comprises the password encrypted and a 
SIG(u>TB) signature. Once the reply is received, the client can generate 

the basic key B and the truncated basic key TB. The client 
Thc firewall then generates a reply whose elements are foen encrypts the password P, expressed as 
shown above the left arrow in FIG. 22. The reply comprises 
the destination basic public key ID, the cipher method, key 65 ENC(n, * < ^ 

method and md method, encrypted session key and the The signature is generated using the md method on the 
signature. combination of (he contents of the original request sent to 
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the firewall as shown above the right arrow in FIG. 22, 
represented as T, the cleartext password P and the truncated 
basic public key TB, as expressed as 

5 

SKS(T+P+TB) 

The encrypted password and the signature are then sent to 
the firewall. The session key exchange then completes and 
session data communications can begin. 

While the invention has been described with respect to a "» 
limited number of embodiments, it win be appreciated that 
many variations, modifications and other applications of the 
invention may be made. 

What is claimed is: 

1. A method of inspecting and selectively modifying 
inbound and outbound data packets in a computer network, 
the inspection and selective modification of said data pack- 
ets occurring in accordance with a security rule, the method 
comprising the steps of: 

generating a definition of each aspect of the computer 

network inspected by said security rule; 
generating said security rule in terms of said aspect 
definitions, said security rule controlling at least one of 
said aspects; 

converting said security rule into a set of packet filter 
language instructions for controlling an operation of a 
packet filtering module which inspects and selectively 
modifies said data packets in accordance with said 
security rule; 

coupling said packet filter module to said computer net- 
work for inspecting and selectively modifying said data 
packets in accordance with said security rule, said 
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accordance with a security rule, where each aspect of said 
computer network inspected by said security rule has been 
previously defined, said security rule being previously 
defined in terms of said aspects and converted into packet 
filter language instructions, a method for operating said 
security system comprising the steps of: 

providing a packet filter module coupled to said computer 
network in at least one entity of said computer network 
to be inspected by said security rule, said packet filter 
module implementing a virtual packet filtering machine 
inspecting and selectively modifying said data packets 
passing into and out of said computer network; and 
said packet filter module executing said packet filter 
language instructions for operating said virtual packet 
filtering machine to either accept or reject the passage 
of said data packets into and out of said computer 
network and to selectively modify said data packets so 
accepted. 

10. The method according to claim 9 wherein said aspects 
include network objects. 

11. The method according to claim 9 wherein said aspects 
include network services. 

12. The method according to claim 10 wherein said 
aspects include network, services. 

13. The method according to claim 12 wherein said object 
definitions include the address of said object. 

14. The method according to claim 9 wherein said virtual 
machine performs a data extraction operation. 

15. The method according to claim 14 wherein said virtual 
machine performs a logical operation. 

16. The method according to claim 15 wherein said virtual 
machine performs a comparison operation. 

17. The method according to claim 9, wherein said 
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packet filter module implementing a virtual packet 3* selective modification is chosen from the group consisting of 
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filtering machine; and 
said packet filter module executing said packet filter 
language instructions for operating said virtual packet 
filtering machine to either accept or reject the passage 
of said data packets into and out of said network 
computer and selectively modify said data packets so 
accepted. 

2. The method according to claim 1, wherein said aspects 
include network objects. 

3. The method according to claim 1, wherein said aspects 
include network services. 

4. The method according to claim 2, wherein said aspects 
include network services. 

5. The method according to claim 4, wherein said object so 
definitions include the address of said object. 

6. The method according to claim 1, wherein the filter 
language instructions of said step of converting are in the 
form of script and further comprising a compiler to compile 
said script into said instructions executed in said step of 55 
executing. . 

7. The method according to claim 1, wherein in both said 
steps of generating said aspects of said network and of said 
security rule are defined graphically. 

8. The method according to claim 1, wherein said selec- 60 
tive modification is chosen from the group consisting of 
encryption, decryption, signature generation and signature 
verification. 

9. In a security' system for inspecting and selectively 
modifying inbound and outbound data packets in a computer 65 
network, said security system inspecting and selectively 
modifying said data packets in said computer network in 



encryption, decryption, signature generation and signature 
verification. . 

18. In a security system for inspecting and selectively 
modifying inbound and outbound data packets in a computer 
network, said security system inspecting and selectively 
modifying said data packets in said computer network in 
accordance with a security rule, where each aspect of said 
computer network inspected by said security rule has been 
previously defined, said security rule being previously 
defined in terms of said aspects and converted into packet 
filter language instructions, a method for operating said 
security system comprising the steps of: 
providing a packet filter module coupled to said computer 
network in at least one entity of said computer network 
to be controlled by said security rule, said packet filter 
module emulating a virtual packet filtering machine 
inspecting and selectively modifying said data packets 
passing into and out of said computer network; 
said packet filter module reading and executing said 
packet filler language instructions for performing 
packet filtering operations; 
storing the results obtained in said step of reading and 
executing said packet filter language instructions in a 
storage device; and 
said packet filler module utilizing said stored results, from 
previous inspections, for operating said packet filter 
module to accept or reject the passage of said data 
packets into and out of said computer network and to 
selectively modify said data packets so accepted. 
19. The method according to claim 18 wherein said 
aspects include network objects. 
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20. The method according to claim 18 wherein said 
aspects include network services. 

21. The method according to claim 19 wherein said 
aspects include network services. 

22. The method according to claim 21 wherein said object 
definitions include the address of said object. 

23. The method according to claim 18, wherein said 
selective modification is chosen from the group consisting of 
encryption, decryption, signature generation and signature 
verification. 

24. Id a security system for inspecting and selectively 
modifying inbound and outbound data packets in a computer 
network, said security system inspecting and selectively 15 
modifying said data packets passing through said computer 
network in accordance with a security rule, where each 
aspect of said computer network controlled by said security 
rule has been previously defined, said security rule being 
previously defined in terms of said aspects and converted 



26 



into packet filter language instructions, said security system 
comprising: 

a packet filter module coupled to said computer network, 
said packet filter module operating in accordance with 
said security rule, said packet filter module implement- 
ing a virtual packet filtering machine inspecting and 
selectively modifying said data packets passing into 
and out of said computer network; and 

processing means for reading and executing said packet 
filter language instruction integral with said packet 
filter module, said processing means operating said 
packet filtering module to either accept or reject the 
passage of said packets into and out of said computer 
network and to selectively modify said data packets so 
accepted. 

25. The method according to claim 24, wherein said 
selective modification is chosen from the group consisting of 
encryption, decryption, signature generation and signature 
verification. 
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